Data visualization for Inspect AI large language model evalutions.
Welcome
Welcome to Inspect Viz, a data visualisation library for Inspect AI. Inspect Viz provides flexible tools for high quality interactive visualisations from Inspect evaluations. You can author visualisations in any Jupyter Notebook then include them in documents as static images or or websites as interactive Jupyter Widgets.
Getting Started
First, install the inspect_ai and inspect_viz package from GitHub as follows:
Then, read on below for a tutorial on the basics of Inspect Viz. Once you are up to speed on the basics, check out the Examples and read the Components articles for a more in depth explanation of the library’s features and how they fit together.
Plots
Tip
Inspect Viz is built on top of the Mosaic data visualization system which is in turn built on Observable Plot.
Below we’ll describe the the Inspect Viz Python API, which typically maps quite closely to the Observable Plot JavaScript API. Once you start creating your own plots and are using Google or an LLM to help with development, asking how to do things in Observable Plot will typically yield actionable advice.
The x axis for this plot is not mapped to a column, but rather to a count() transform ( transforms enable you to perform computations on columns for plotting). The fill option gives each species it’s own color. We also specify that we don’t want a y_label (as the species names serve that purpose) and a smaller than normal height.
Tables
You can also display data in a tabular layout using the table() function:
from inspect_viz.table import tabletable(penguins)
You can sort and filter tables by column, use a scrolling or paginated display, and customize several other aspects of table appearance and behavior.
Filters
Use inputs to enable filtering datasets and dynamically updating plots. For example, here we add a select() input that filters on the species column:
Note that we set the opacity of the dot mark to 0.1 to help mitigate oversaturation that results from large numbers of data points being stacked on top of eachother.
Marks can also be used to draw lines, arrows, text, or images on a plot.
Params
As illustrated above, inputs can be used to filter dataset selections. Inputs can also be used to set Param values that make various aspects of plots dynamic. For example, here is a density plot of flight delays which uses a slider() input to vary the amount of smooth ing by setting the kernel bandwidth:
Apply the bandwidth to the plot (plot automatically redraws when the bandwidth changes).
Selections
Above in Filtering we began exploring dataset selections. Inputs are one way to set selections, but you can also set selections through direct interaction with plots.
For example, below we stack two plots vertically, the dot() plot from above along with a bar_x() plot that counts the sex column. We then add an interval_x()interactor that enables us to filter the dataset using selections on the dot plot.
There are a number of new things introduced here, click on the numbers near the right margin for additional explanation.
A Selection is a means of filtering datasets based on interactions. Here we use an “intersect” selection for application of a simple filter from dot plot to bar plot.
2
The range selection is set via the interval_x() interactor (which enables using the mouse to select an x-range).
3
The Brush defines the color of the interactor (in this case #888, a medium-gray).
4
The range selection is consumed using the filter_by parameter.
5
We set the x_domain for the bar plot to “fixed” so that the scale doesn’t change as the dataset is filtered.
Try using the mouse to brush over regions on the dot plot—the bar plot will update accordingly.
Data
In the examples above we made Data available by reading from a parquet file. We can also read data from any Python Data Frame (e.g. Pandas, Polars, PyArrow, etc.). For example:
import pandas as pdfrom inspect_viz import Data# read directly from filepenguins = Data.from_file("penguins.parquet")# read from Pandas DF (i.e. to preprocess first)df = pd.read_parquet("penguins.parquet")penguins = Data.from_dataframe(df)
You might wonder why is there a special Data class in Inspect Viz rather than using data frames directly? This is because Inpsect Viz is an interactive system where data can be dynamically filtered and transformed as part of plotting—the Data therefore needs to be sent to the web browser rather than remaining only in the Python session. This has a couple of important implications:
Data transformations should be done using standard Python Data Frame operations prior to reading into Data for Inspect Viz.
Since Data is embedded in the web page, you will want to filter it down to only the columns required for plotting (as you don’t want the additional columns making the web page larger than is necessary).
Data Selections
One other important thing to understand is that Data has a built in selection which is used in filtering operations on the client. This means that if you want your inputs and plots to stay synchoronized, you should pass the same Data instance to all of them (i.e. import into Data once and then share that reference). For example:
from inspect_viz import Datafrom inspect_viz.plot import plotfrom inspect_viz.mark import dotfrom inspect_viz.inputimport selectfrom inspect_viz.layout import vconcat# we import penguins once and then pass it to select() and dot()penguins = Data.from_file("penguins.parquet")vconcat( select(penguins, label="Species", column="species"), plot( dot(penguins, x="body_mass", y="flipper_length", stroke="species", symbol="species"), legend="symbol", color_domain="fixed" ))
Crossfilter
In many cases you’ll want to have an input or interactor that both consumes and produces the same selection (i.e. filtered based on interactions with other inputs or interactors, but also able to provide its own filtering).
Inputs
This example demonstrates crossfiltering across inputs. We plot shot types taken during the 2023 WNBA season, providing a select() input that filters by team, and another select() input that filters by player (which in turn is also filtered by the currently selected team). Click on the numbers at right for additional explanation of the code.
Create a crossfilter selection, which enables inputs to both consume and produce the same selection (conditioning their available choices on other inputs).
2
The team select box targets the filter selection (filtering both the choices in the athelte select box and what is displayed in the plot).
3
The athlete select box is both filtered by and targets the filter selection, enabling it to both confine itself to the selected team as well as filter what is displayed in the plot.
4
As different teams and players are selected, the y-axis may take on differnet values and ordering. These options ensure that the y-axis remains stable across selections.
Interactors
This example demonstrates crossfiltering across plot interactors. We plot histograms showing arrival delay and departure time for flights. When you select a range in one plot, the other plot updates to show only the data within that selection—and vice versa. This bidirectional filtering is achieved using Selection.crossfilter(), which ensures each plot’s selection affects all other plots except itself. Click on the numbers at right for additional explanation of the code.